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ABSTRACT | 


Reported is an evaluation study involving learning _ 


disabled (LD) children (6 to 13 years 01d) in 23 LD programs in which 


children with successful outcomes were compared to children with 
unsuccessful outcomes. Among conclusions discussed are that. Ss in the 
successful outcome group did not differ from those in the 
unsucc2ssful outcome group in age, ‘ability level, or academic 
attainment when they entered the LD program; and that the 
unsiccessful group displayed more specific disabilities which were 
rated as moge severe than those of the siccessful group. It is noted 
that the formative evaluation model discussed in the document 
considers the interaction of programmatic components with each 
child's individual; characteristics, covering antecaient, transaction, 
and outcome variables. (IM) 
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SUCCESSFUL OUTCOMES IN LEARNING DISABILITIES PROGRAMS: 
A FORMATIVE EVALUATION MODEL 


ABSTRACT ' 
. ae F 


’ 


Formative eValuations which assess program prpcesses and outcomes 


of learning disabilities intervention programs, with the aim to modify 
\ ; 


-_program components, are necessary if programs are to provide appropriate 


services for learning disabled children. | These evaluations should con- 


siders antecedent, transaction, and outcome variables of the inter'vention. 


’ 


This paper’ presents a model for a formative evaluation which considers 


) 


« . 


these variables. An evaluation study, done by the authors, which com- 


pares children with successful outcomes v. children with unsuccessful 


outcomes in 23 learning disabilisAes programs is discussed within the . 


_f : 
evaluation model framework. : P 4 
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‘AZ CHILD VARIABLES: 
1) demographic 
2)nature and seve- 
“rity of disability 
3) 1Q and achievement 
4) Personal/social 
behavior: 
“regular class 
-resource class— 
4) Self-concept 
_6)Task-related 
behavior: 
' regular class 
.-resource class 


B. PROGRAM .VARIABLES: 
1)Teacher variables: 
--demographic 
-experience 
-training ’ 
2)Curriculum variables: 
-subject 
-type and scope 
3)Setting variables: 
-teacher-child pratio 
-location of service 
ayeerertel/aiagncet ie 
variables: 
, type of aiaeiourees 
used 
-people involved 
-types and amounts of 
information to 
teachers 
-placement .errors ° 
5)Support system 
_ variables: ~ 
--parent involvement 
-regular teacher 
involvement 
-other involvement 
6)Resource allocation 
. variables: 


‘ 


-method of allocation. 


variable 


-per. pupil expenditure 


-% allocation per pros am, 


* 


Table 1 


1)Program intensity: 
-sessions per week 
-program duration 
-amount of. teacher- 
child contact 


2)Quality of LD teacher- 
child interaction: 7 
~-nature of reinforce- 
ment schedules | 
-rating on hostility- 
acceptance dimension 
"rating on autonomy- 
‘control dimension 


3)LD teacher perception 
_of probability of 
program outcome for 
a child 


\ 


4)LD teachipg methodolog 


\ Formative Evaluation Model for LP Intervention Program 


a { 
ANTECEDENT VARIABLES , TRANSACTION)VARIABLES | OUTCOME VARIABLES 


1)Achievement outcomes 


2)Personal/social 
behavioral dutcomes: 
-regular. clafs 
-resource class 


3)Task-related beha- 
vioral outcomes: 
-regular class, 
-resource .class 


> 


4)Progress on specific 
disability 


5) Self-concept out= 
comes 


6)Perceptidn of causes 
for program outcomes: 
y‘-LD teacher’ 


used: -regular teacher 
-instructional “child 
-planning -parent 
-assessment -déther 
+flexibility of method 
5)Lesson properties: 
-continuity ' . ' 
-amount of teacher 
involvement . 
-distraction factors 
) 
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SUGCESSFUL OUTCOMES IN LEARNING DISABILITIES’ PROGRAMS : se 
A FORMATIVE EVALUATION MODEL , 
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“INTRODUCTION . 


any educators, involved in special education intervertions for 


lea ng disabled children, have stated a need for a logical. formative 
eva duation strategy to assess the impact of theiz programs or to sua 


Aformation for improvements. The questions of what factors shguld be 


ff 


fexemtoed and what sipthadologies should be chosen once the factors are 


established ‘are difficult ones. The ‘bene of this Paper As. on one type 
-— ‘ -: fe 
of formatiye evaluation model and a study #hich utilized. that model. , _ 


Scriven (1967) describes formative evaluation as an” a appraisal of the 


- 


value or worth of a process or product which gives decision-makers oar 
formation as to which components of the visti tipui need.modification.. 
Sanders and Cunningham (1973) discuss such an evaluation strategy as an 
“interim evaluation" which works. with pieces of the program to upgrade the 


overall intervention. For our purpose in this paper, evaluation will |, 4 


refer to a formative assessment of program components, the processes and 
. ¢ 
prodycts, which can be revised. i 


, \ 5 
categorization of program factors, adapted from a model by Robert 


_ /Stake (1967), provides a framework to ee program variables to be 


\ ' u - 
‘ assessed. Antecedent variables include those with-in program factors which 


exist prior to the actual involvement of the child in the LD intervention 
program. Examples of antecedent ‘variables include the child's IQ level 


or the teaeher's number of years of experience. Some evaluators refer to 


these variableé as "incoming variables" (Hulka, 1975). : 


a Hy 


- 


1the study was funded by the North Carolina State*Department of 
Public Instruction under 1973 state legislation, The Act to Provide Public 
Education to Children with Learning Disabilities. 
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a (if . ee ke 
- variables. . An,example’ of an outcome variables includes es of 


bles in each of the three categories would likely be. impossible for a 


r 


“ 


‘Secondly, transaction variables are those program factorg which des- 


cribe what is actually happening in the educational intervention with the 


child. Examples of transaction variables ‘include the actual number of 


‘hours a teacher spends with the child per week or.the teacher's use of 


particular reinforcement/é¢hedules with the child. The paucity of research 


or information on transaction variables in intervention programs is acute. 


/ 
4 


(Karnes, 1976). 


The third category of program factors to be examined is ottcome 


progress made in an area of academia work focused upon by the program. 
, 7 * . / 
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These variables have traditionally been given the most attention—-a 


‘ e 
shortcoming true of most evaluations of educational projects (Cain and \ 


Hollister, 1972). A focus on outcome variables:alone has the unfortunate . 


result of producing little knowledge about what fostered or inhibited 
a positive outcome, and consequently has less utility for formative 
assessment. a ‘ | 

Table one flluetnetes a hypothetical evaluation model in which 
critical variables in an LD intervention program are displayed according ~ ° 


to an adaptation of Stake's model. An assessment of all of these varia- 
‘ 


single program or a single school system. However, it is increasingly j 


e e 
more apparent that evaluations of LD intervention programs should be 
\ +! rd 
| 
conducted based on a framework of antecedeat, transaction, and program 


> 


outcome variables. The authors believe that data can be obtained from 


] 
LD, programs and ,formative assessment methodologies can be utilized to 


¢ 
~~ 


evaluate these programs. 


(Insert Table 1, page 3) Ms eS \ 
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The following study, doné by the authors, is used as an example of a| 


formative evaluation study which has variables that can be organized accor- 
: ( 
ding to the model in Table 1. Some of the variables listed in Table 1, 


ji ‘ 5 : 4 4 Fi 
though recognized as important, were either not applicable ‘because of the 


2 group comparison design, .or were beyond the scope of the study. 


STUDY BACKGROUND AND PURPOSE ° 


' : . This study, sonia by the Frank ‘Porter Graham Seidl Beyectopasae: 
Center at the University i tioceCane tua te Chapel Hill, focuged on th 
Categorical Learning Disability Programs in North Carolina during the second 
year of the pconvanits effort. These programs, of which there were 89, 


e: | opérated primarily on‘an itinerant resource room format. The ‘children 
: ego 
% served remained in their regularly assigned classrooms and reported to the 


. . resource room at regular intervals for special instruction. Each resource 


“8 teacher made the referrals almost exclusively and the programs' learning 
\ : - 
disabilities teachers were almost solely responsible for assessment and 
placement of ,these children. 


. ’ 


eee af Children in these programs were mostly boys @% boys to 1 girl) and 
ee usually white (4 white to 1 nonwhite). . They ranged in age from 6-13, 


- but most of the children were between 9-12 and in the 4th grade. 


id 


The study was ‘designed as a formative evaluation of the overall 


» programming effort. fatten to identify the factors which contri- 
> 4 
buted’to a successful or unsuccessful educational’ outcome for the par- 


- 
/ 


ticipating students. Major que tions explored in the study were: 


i] ‘ 
- ’ 


Bee Do antlacen who show fhe most successful outcomes 


; : a UE in the categorical LD program differ on the fol- ‘ 
? : lowing characteristics from those who show the least 
ras rz succes$ful outcomes?” 


. - 
- ‘ --nature of thats 7 and educational. problems~ 
. ’ = we 


) 


“a 


- 


~severity of their aries and ‘educational 
probions 


. -denographic characteristics 

he ao _ 

Do the following program components differ for 

the group with successful outcomes and the group: = 
with successful outcomes? j 
--setting variables 

--referralsystem and diagnostic processes a . ; 
support systems variables 

--curriculum materials and instructional methodologies | : 


--program teachers’ perception of causes of program gubeones 
\ 

METHODS ’ , x 

Sample. Each teacher: of the 44 Categorical Learnin Disabilities 
programs, which were in the second year of operation, was sent an initial: 
screening quent toanadve:. They were asked to nominate the 5 children for 
_whom their program had been the aoRt successful and. the 5 children for 
whom the een had been di least successful. Four criteria items were 
recommended as the basis of those nominations: academic success, sic 
skills success, social adjustment success, and personal en success. 
Based on the intormatién obtained, 23 programs representing a state-wide 
sample were chosen for further study. To narrow the sample to a vorkable 
size, one child, from each of the two grdtpe: at 5 successful and 5 unsuc- 
cessful nominated, was chosen at random for the ineadias compaetoas, The 
eeeinataas population, eeeste consisted of 23 children in the successful . 
—e group (1 from each program selected) and 23 children in the unsuc- 
cessful outcome group (1 from each program melectedys a 


“Procedure. Data collection techniques ancluaed dieetien suReENaSHE 


at the program teachers and examination of the students records for each of 
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‘ 


the\23. programs. In the interview period of 2 1/2 hours, equal time wase*, 


spent. in discussion of the specific program for the child with the suc- 
eS 5 a Fi e : y Z 7: 
, ; cessful outcome and for the child with the unsuccessful outcome. Mea- 


wwe 


,Sures were taken to insure the confidentiality of data from each interview 
oy , = aa + | 
made and each récord examined. The interview questionnaire focused on the ~ 3 


major areas of child tharacteristics and program’components listed on : “ 


page 4-5. 


: °° RESULTS 4. rs , . 
: : e . - 0 td ae. 

In examining the LD programs of children who had successful out~ 

bet comes versus those programs of children with unsuccessful outcomes, both 


a 
cases evidencing differences and'those evidencing no differences were : 


informative. 


. S 
Antecedent Child Variables | 


f we "Age and ability level (IQ). ‘The average chronological age for the 


successful group was 112.17 months,and that for the unsuccessful group was” 
a . 


\ : 
107.34 months. The average IQ for the successful group was 95.5 and for 
the unsuccessful group was 90.60. A group (2) X grade (3) analysis of 


variance for each variable did not yield significant differences between 


the two groups in either age or IQ level. 


- 


Initial achievement level. The average grade equivalent score, as 
eee ooo . 


measured by norm-referenced tests given at thé beginning of the year, 

. . was 2.72 pie aie successful group aut ae for cheunadecessful scouts The 
successful group's average grade level was 3.8 and the dnedoewketel group's 
average grade level was bh, Although the analysis of vaetnads on. these 
scores showed an expected grade level effect (F=10.29, df=2/33, p<.001), 

no significant differences were Pie eean the successful and gee 


successful groups in initial achievement level. 


. 
. F F = 
2 + 
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Severity of learning disability. In order.to obtain a measure of 


. . 


the number, kind and severity of each child's’ disabilities, ‘the LD teachers 


were asked to complete a 48-item learning disabilities’ rating sdale (McKinney, 


1975), The scale provides ratings on a 5-point continuum for each of. 


the following lheareas: eo es : 


1. 


2. 


10. 


-ll. 


- Orientation--judging time, spatial orsentation, judging 


Physical development--hearing, visual.acyity, eye 
control, speech, and muscle control. 


Self help--using anette, eattog, dressing, and 
toilet training. , 


. Motor development—-coordination, balance, and manual 


dexterity. 


. Visual perception--discrimination, figure- propia, and 


visual/motor integration. 


. Auditory perception--discrimination, following direc- 


tions, comprehension and receptive vocabulary. 
. & 


. Memory--retention of auditory information, recall of 


non-meaningful and meaningful visual PHEDEDEE OT: and . 
retention of words. : 


. enoken language--senterice structure, expressive 


vocabulary, relating experiences, ideational fluency.. 


. Conceptual ekills--abstraction, creativity, logical 


thinking, conceptual tempo (reflective v. compulsive). 


° 


relationships, and. learning directions. 


Personal/social behavior--cooperation, attention, coping 
skills, social acceptance, responsibility, worksattitude,, 
tactfulness, and fexibility. 

Ra! 
Specific achievement--reading, writing, spelling, and 
arithmetic. 7B 


The teacher was asked to describe the child's disability in each 


area (where.need for remediation was évident)as either mild, moderate, 


sévere, or profound in severity. In each case ea was based 


\ 


on a)the number of specified defacits displayed in that nore b)the extent 


which the deficits in a given area contributed to davadopbenee dais: 


pis <2 


. 
ad ‘ s 
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Cad 


‘ ‘and c¢) the amount of time devoted ‘to remedial work in that area. In | 


os e mapet son to the severity ratings far each of the. Il subscales, a general 


o gsetty index was computed by multiplying the number of diaabilittes 


» y 
by the average severity rating. q 
‘ : J : ° 
Differences were apparent in both the number of disabilities _ 2 


that were, identified and in the average severity of the disabilities 


& 


e that were rated. Children who were classified as successful ia i 
% 


-an average of 5. 91 specific disabilities Sonpared to 7.83-for desig who 


_ were rlasedtted: as iiendenestat F = 9, 12, p<. 004). ‘Children who were ! 


Py * é « 


described as “successful; earned a méan severity, rating of 2.15 ‘compared ‘to . , 
2.83 for those’ who were described as unsuccessful (F'= 14.55, p<.007). ). 


Consequently, eitheres in the successful group shamed a significantly 
lower LD a Sndex es 85) compared ‘to those in the unsuccessful Pee 


_ group (22, 94): (F = 11. 39, p<.002). Table 2. displays this data,’ 
In summary, these findtioes, indicate that in boththe- number of and 
severity of disabilities, the unsuccessful children were rated as signi- 


ficantly more disabled than were the successful children. \ 


7 : = . (insert Table 2, page 9) 


#Antecedent Program Variables 


‘Referral system. The primary referrer was, for beth groups, ‘the ot 


regular classroom teacher. Few other peOple were involved in the referral 


’ 


process.. The quantity ang ‘type of referral or diagndstic information 


: I 3 
available to the LD teacher was the same upon entrance tothe program 


¢ 


for both groups. | elo . | . 


“ 


' , Diagnostic process. Psychologists were used in this process 


, 


primarily to administer individualized IQ tests. Many teachers indicated 


if. **" 


Ee te 3 4 a agit, 4 —_ - ‘Table'2 


ae a ‘Multivariate Analysis of Variance of the LD Severity Index . 


Physical development 


Self help 


Motor development 


Visual perception 


Auditory Perception “ 


Memory <— 


Spoken Language | 


Conceptual Skills 


Orientation 


Personal/social behavior ' 


Specific Achievement 
\ 


that this 19 information arrived, however, too late or was too obtuse to . 
be helpfui in placement oF p planning decisions concerning the child. 
“Reports from regular | class EeBenere were available in about half the cases, 
We . * Aibue they. were generally veueeted to be too informal aid unspecific fe be 
.’s useful. Twenty-six bexcene of the ‘successful outcome children were vacaveed 

. “o . ay having been misclassified. as teeing disabled, ii ‘while 35% .of the 
i o 
es unsuccessful outcome chitdren were so reported. kd successful children. 
‘ were most often referred to as having a "mild senate problem" while the ° 

unsuccessful divider wast often were referred to as being "emotionally . 
-_ disturbed,” No notable differences in the type or duanel ey of dtdgnoette 
| activities were found for the two groups. | 7 

Supgort systems. Thoust the incidence of regular teacher ers 

pation in the child's LD program for either econ is low, regular teachers 
were twice as likely to be involved in the case of the successful outcome 
child. The same proportion .was true of parent involvement. Other per- 


i 


sonnel input into the LD programs, efther in-planning or participation was 
‘ . ’ » - . - 
t 
negligible. 
Setting variables. No differences were noted in the teacher-child 


aie ratio between the successful and unsuccessful. groups. 
a. 


Transaction Variables 
BEN Program intensity. There,was no significant differences between the 


‘ _ two groups for the average duration of tre%tment. However, children in the 
ie '™ guecessful grotif-were seen more frequently (3.8 hours/week) than weré those 
a . dn the unsuccessful group (2.9 hours/week). This finding represented a 


‘significant differeiice (F = 4.26, p<.04). 


eet e \w ‘teachers' instructional sisthtotopll One of the major findings‘ 


4 


Ra a this study was that, for the groups who were ‘succeeding and not succeeding, 


a "age , a 


‘ 


# , rm Com ‘ 
the methods sabi in teaching both groups were basically the same: 
a = Reading was the main thrust of the program for both 
; groups. 


2 


=f 2. An almost exclusive auditory-visual apptoach was used 
a in the teaching of reading. More auditory stress’ was 
Ne. a ee .. found-for the successful group and more visual stress 
O49 e was found for the unsuccessful sche 


* ; 3. Kinesthetic-tactile approaches were rarely used and, 
fg) Ags 5 when used, were: qupplesiental. 


4. Few teachers altered their method strategies: for the 


unsuccessful children even after assessing the child's — 


progress. - Those teachers who said they didvary methods 
ia only mrrera the quantity of the work. 


Se Selection of sheatiars for the child in either group 
was based on familiarity and availability rather than 


( et 
Gutcome Variables 


Achievement outcomes. Children who were classified as successful : 


‘made significantly more academic progress than the children classified 


« a 


as unsuccessful (t = 2.47, p< .02). Those who were -classified as un- 


successful failed to show progress. 


LD teachers’ perceptions of causes for program outcomes. An 


' efficacy rating by the LD teachers concerning that part of the program 
they felt contributed most to its success when successful and’ its lack 


of suctess when unsuccessful yielded’ the following results; ~ 
+ e - . 5 * 
we ‘ 1. Teachers reported that instructional methods used 
’ with successful outcome children contributed most 
to the success of that group. 


2. Teachers reported that tthe child's motivation and * 
learning style was the biggest barrier to success 
E -.; in the unsuccessful outcome cases. ; 


' These ratings are noteworthy because instructional mogheds for 


; both groups: were basically the same. In other words when the. child in, 


‘the program succeeds the teachers take credit for the. positive outcome} 


oid 


‘ . ot : 


ina dtagnostic-perscriptive or task-analytic fashion. “ Ls 


Ie aay 8 5 . eo $ on 
ae af . ° ; . e 
i a 
' . ‘ * 
’ ‘ . . ‘ 
‘ a * s 
‘ . 
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but when the child fails, the child is seen to be at fault. , 


CONCLUSIONS - © og 
In summary , children in the Successful outcome group did not differ 
from thoae-chiléred in the siadccasstot outcome group in age, ability level, a. ed 
. or academic attainment when they entered the LD program. The unsuccessful 
group did, however, display more specific disabilities which were rated. | 
“ - % as more severe than those of the successful roup, by the peuhee. By the 
ffend of the year's intervention, the successful group had made significantly 
more academic progress chit the unsuccessful group. om 
An esanination of the program variables, to determine if the programs 
; eve different for roaee groups, found no differences in suerisukon, 
. setting, referral, diagnostic, or support degen variables. Program 


: intensity, the amount of time per week the child was in the program, did / 


‘vary significantly for the two groups with the successful children avera- 


os 


ging 1 hour more per week than the unsuccessful children. Whether the 
, — 


successful children progressed because of the added instruction or whether 
the teachers chose to’ spend more time with the children evidencing pro- 
gress was not determined by this study. 

The absence of instructional methodologies form fit to a child's 


specific disabilities and the notable lack of changes in methods’ once the 


unsuccessful ‘children began to fail are probably important, contributing - 


x 


“a 
‘ 


~ factors to these children's lack of success. 


DISCUSSION : : sei 
Se | 

An effective learning-disabilities program contains anentire system 
of programmatic components which interact with the child's unique charac- 


teristics. The formative evaluation model discussed in this paper attempts 


15 ? se a 


€ ‘ 
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w 


, to consider that interaction with its antecedent, transaction, and outcome 
“ 
variables. Without continued, systematic’ evaluation of components in each 


of these categories, program directors ane other decjsion-makers will not 
' 
be able to create Nes eeiegat services for LD dutitren: 


Additionally, an oT point here, and one often neglected, is that 


_ formative evaluations need tox focus on Pies which fre modifiable to be . 
ae 


, 


, trully relevant’ to the ae process. Once a factor is found 


to be correlated with program success or failure--what can be done ‘abahit it.” 


4 
Cae 


haponee the decision question. This model would insure ‘that program out- 


4 e a = 


, 


comes are seen in the context of program Kenianeionel mi therefore, more 


| 
' 


knowledge about modifications would be available to the decision-maker. 
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